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5 ate offered to assist the development of an 
evaluation model, includ : 
terminal objectives and 
protocols, Checklist questions cover the following areas: (1) goal 
relevance, (2) procedural instruction, (3) organization and se-1ence, 
(4) stimulus function of didactic units, (5) response funetion of 
didactic units, (6) integration of program, (7) motivation level of 
program, (3) program language, (9) external factors, and (10) ; 
relations between terminal objectives and single didactic objectives. 
Examples of various kinds of unit charting protocols also are 
provided. (EMH) 
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Iewen at'the progent stim of our browludae, much time and work 
in the post-writing stage could often be saved by a more systumatic exami- 
nation uf the progranwoed raateriol. before the cmpivical tests. The fact 
that the successive wimypirienl tev-our of the mercerials are the final and 
Most lnportant steps in the evaluation proce + devs not miean, of caurse, 
that we have to rush into premature data cellection with hastily produced 
and unpolished versions of the vtudy material, ‘hile an arbitrary reading 
through of the material tosy be fairly inefficaent, a systematic tapping 
of costain apparent chosacteristies of the material on the othee hand, me ay 
be very fruitful, We misht call this exami nol apperent character 
the aes of the “'pheno-struciure’ of didacite sequences (as ‘distinguished 
efare! thatmight be enarried out during , 


In the present paper several aids for this mapping process will be 
described and discussed, melucding checklists, dingrams of the rel:tions 
‘butween turmirnal onje “tae und the single disactie units, and various 
kinds of unitechartsny pr SS It is felt thet the oxamination is considerably 
facilitated by summarizing mids of these kinds. The risk of forgetting some 
important aspect of the examination is diminisned, and a long series of 
questions about the material is nich easier to anuwer on the basis of a 
summarizing protocol than on thy basis of the non-aided looking through of a 
long sequence of material, in which the different parts cannot be kept within 
the field of sight at the seme tir 
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‘ al note: In this paper the term "didule" is 
didactic unit of a programmed mat: is 
thought of as the program ‘s contribu ion to the basic unit of interaction between 
pupil and study 7 al, cove 2 rule three components: the informational 
component (termed "'stimule"'), the request for oe (tex 
and the feedback component (termed pooreule |. e English term tframa!! has 
often been used in the same sense as the present foes "didule". However, 
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lary prompting ocen avoided? : 

e@ Tele baa metningful prompting (by means of logical induction, 
peritlel oternplification ete.) bes chosen where appropriate rather’ 
than ueaeceisary formal prompting of a kind that may distract the 
student “5 attention and divert it to unimportant aspects of the 
coniumunication 7 
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Aga rule, can the questions be answered on the basis of the informa~ 
tion thet the student has received? 7 
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without understanding of the content of the didule (by formal- linguistic | 
pattern coinpletions, ute, )? 
Has the response re quest been designed so as to give a reagonable | 
guarantee that the student has understood the wessential communi- | 
cation of the particular ciciebe? In other words! Are the response 
requests colevant to the ain. of the particular didule, (always as ssuming | 
that enech didule has a spec iin able aim)? 
Goes tie grogram activate varyin, types of response behavior 
(writing, drawing, caleulating, comprring, ete.) ? 
Ge the alternatives cf # multipleschoice question represent all 
reasonnble, nen-trivial sources of misunderstanding? In other | 
words: Lu it ‘difficult to think Gf typical student errors not included | 
amone the w.ternatives listed? 
Are the alternatives in a multiple-choice question so designed that | 
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w= bear made interesting and relevent te the experiences and neeas of 
the particulir  tudent groups ? 

Do the pos sibilithe jaf individuclization within the program (available 
branching, arriigerients) correspeod closely cnough to the results (ot 
the initial aastyses of the target papulaticn, 50 that, for instance, 
students with conuiderabl. previous knowledge of the subject matter 
are not forced to oo Urrourh the same mates ial as students Without any 


mg povnible 


previous knowledge? 


Examination of the 


Do the leves of vecsbulary ane the structure of .sentences ccerm to be 
sufficiently well auapted te the linguistic habits of the particular student 
group so as net to be an unnecessary barrier to communication of the 
main contents? 

Ts the meanina ef new terms defined oF otherwise demonstrated clearly 
enough ag sour o9 they are introduced ? 

Io the language of the prograrn always clear and unambigous? 

Does the language always have an eptime! degree of precision as judged 
from the goal statements of the program (that is, neither with too low 


ermal form of the program: Language 


a degree of precision, incuuding sloppy everyday terms where precision ” 


is needed, ner with an unnecessarily high degree of precision, sgrh as 
various scientific distinctions aot te be further utilized in the desired 
terminal behavior)? , 

Is the use of didule numbering, punctuation, abbreviation ctc, both 


correct and consistent? 


zrnal form of the progran 


Is the general lay-out of the program both 
economically defensible? 

Iu the typography sufficiently clear for the intended student group, and 
also designe’ to emphasize things that should be emphasized? 

Are illustrations of various kinds, both within the program proper and 
in possible appendices to be usec as reference material so designed 
technically that they can be expected to result in the best possible 
communication effect? : 
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3, Exainination by f means of didule protocels 


“A didule protocol is a device which makas it ¢asier to make the 
examination both cumplete and systematic, Exaraination by tweens 
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of 


general checklists aids the examiner in coveriny: many aspects, but it 
may not by itself help.him tu cover every single part of the program in 


a-systematic way. As a rule:a didule protocol includes, for each di 


idule, 


a separate examination column, in which + depending. on the particular 


aim - various kinds of questions may be answered when this parti 


cular 


didulois examined, Three general types of such protocols may perhaps 


. be distinguished: subject matter protocols, mothodological protocols, 


and combined protocols studying both subject matter distribution and the 


migteeOlay te al app ‘onchee aoused, 


(a) Subject matter pouicent _ : 
The main aim of the, subject matter protocol is to, some extent 
similar to thé”aim of the T-D diagrams described abova, viz. to 
+ facilitate the examination of the distribution of the subjact matter 


over the didule sequence, A square-ruled paper.may- be a good starting- 


point. The numbers of the didules are written in numerical order along 
the top of the paper, The various subject matter units that are tobe. ° 
examined are listed down the left-hand margin, For instance, these 


-units:may be single conceptual units or conceptual relations which 


have . 


been listed duripg the pre-writing phase as important key points to be 


cévered in the program, Sometimes 
“gical units that logical units that is, to stu at from a series of 


s it-is mov: natural to use psycholo- 


stimulus ~- ~response connections rather than f. + concepts and conceptual 


relations, Whatever the particular kind of uni 


of check marks in the appropriate cells, we report in which of the 
a particular subject matter unit is treated (cf Figure 2). 
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Among the questions that may be answered with the aid of such 
a protocol, the following may be mentioned: (a) Is each single basic | 
subject matter unit represented to a sufficient degree in the program? 
(b) Are some of the didules totally irrelevant to the ynain. subject matter 
to be covered? (c) Is the treatment of the single subject matter units ; 
appropriately distributed (fer instance, an intensive series of didules 
at the first presentation, followed later on by separate repetition items 
with gradually increasing intervals)? (d) To what extent have attempts 
been made to integrate the separate units into meaningful structures, 
and are these attempts made at optimal points in the sequence (for 
instance, where the separate concepts are sufficiently well treated and 


practised) ? 


“When answe ing the last question, for instance, the examiner will 
study to what extent the same didule column contains check marks re~- 
different subject matter units. It is usually desirable * 

his is th in many didules in order to facilitate integrated 
structures of knowledge, At the same time the cxaminer of the program 


studies at what specific points in the didule sequence the new subject 
t presented. As a rule it is confusing to the student, 
if several new subject matter units are introduced in the same didule, 
An integrating didule therefore usually contains citheroa series of units 
introduced earlier in the program or, onc new unit of sudject matter in 
combination with one or more units introduced earlier. 
ih . “se 

It is sometimes maintained that integration of knowledge may be 
facilitated by specific patterns of didules, such as systematic alternation 
("counterpoint"), systematic overlapping, chronological accumulation, 
or reversed accumulation of temporal behavior sequences. The advan- 
tage of the latter arrangement, for instance, may be that the pupil 
sometimes finds it strongly motivating to have the opportunity of going 
through the final phase early in his training as well as of ‘seeing each — 


training sequence lead up to a clear goal, When the final phase of the "+ 


behavior sequence means @ clearcut, perhaps dramatic demonstration 

of the goal attainment, this.type of sequence may be of particular value. “ 
(This specific Arrangement has often been explicitly recornmended by st 
the spokesmen for the so-called ''mathetical approach", represented by 

Gilbert and his co-workers,) Further research is needed on these and 

other patterns of didules recommended fer integration purposes, In this 
experimentation it is, of course, valuable to be able to examine, by ° : 
means of subject matter protocols, the appearance of the various patterns : 


. under study.’ It is then casy for the examiner tu observe inadvertencies 


and to correct arranyegments which have been left in a less than optimal 
shape during the writing process. (A few examples of typical patterns ‘ 
referred to by the terms used above are given in Figure 3.) ; 


(b). Mgthodolou! sical protoca) 

The purpose of the methodological protocols is to facilitate the 
examination of: various, methedological approaches and make it possible 
to answer systematjcally questions of the following types: . 
(a) Has the programmer got stuck in certain working routines with the - 
accompanying risk that the student finds the program monotonous? 


- (b) Are the educational techniques used of such a kind eae besides 


facilitating the student ’s learning of certain subject matter facts, they 
also contribute to educationally desirable side-effects (such as good 
working habits, povitive study attitudes, etc.)? . 


The protacal sheets will be similar to the ones used for gubject — 
matter protocols, that is, *square-ruled papers with didule numbers | 
along the top. However, queations about the educational approach or 
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Note: [In this figure, numb have been given to subject matter 
units. that make up a conne temporal chain, whereas letters 
have ‘been given to subject matter units that are not temporally. 
related to eacli other. A small «rrow below the didule number 
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(4) The sclutien cannet be found wit! ape een, but can be 
Cu nstructed in gome Wot, waunlty with bre cig of the student ‘sa 

backyr oun! exnerience ond/cr his ability te use logical reas: 

niny: (ex mi athematical cperations 3) (Gapsemaledistal! wource)., —; 


- 


At first sight, a subdivision cfthis type may seem te repres gent 
sume kind of unnecessary and pedantic box-+.rtinyy eaverness, 
However, the cbcervation af "response sovrees" isa very important 
art of the didule exarninntion, and many existing prejrams would 
probably have leche quite! different and been mere sffective, if the 
preprammer hac been mide aware early encugh «f onessided 
techniques vtilizes, The Special termunodluvical labels used ("internal 
explicit! ute.) shoul! ast of course, be taken tec gerivusly, but a 
brief and handy term mey be useful in writin, ut these protccols 


wey 


as Well as in diseuodng these phencemena, 
Some éf thy Sdjvesbede bigs mentioned above can be further sub- 
divided, if desirable, The third catopery, for instance, may be 
divided ‘inte maaimally pry ximal cases (when the answor is te be 


found in the immudiately Preceding didule) aal cither cases (when 
ag! 


e 


the answer is found carer in the prepram). Uupoarate "appendice 
(or "panels!) As response sources Can, when appropriate, be listed 
ne a Separate ecteyory, ete, ; 


Such simple Sade! fives the exemincr yuick information abcut 
tha pessible mor sequence, showing, for instance, whether 
ar not the prourarniner hos used SEN gively “internal- ; i 

% sources (which seeme very often te be the cose In.some p 

programs). Each technique’ mey serve an irapertant, function, The 

interngleexplicit appre th focuses the attuntion cf the student fatal 

the, key puints in th ent peer and sears toobe of paerticular 

- value when new tern fe introduced, The externale, 
preximal mothe ce “nteibut te. ane pradual training: of retention and 
may alng be uses. in the prs oun 4uf inte; ration. In order te ensuri 
valuable educations i gideecffectu (jitooud werkin;: habits ete. ), the 
provraminer will pycbebly « ften find it most pas fitable te use either 
man inter: ile implicit erm nal- listal technique, : : 


ERIC 


to earlier answers and to the key-points of the 
_structure of thedidule? (For instance, how high is the ''density" of informa- 
tion-loaded words?) Etc, etc, (For further illustrations, cf Figure 4). 


In certain cases it might be desirable to calculate some index score 
coveritrg the "educational variation" (the distributicn of didules with respect 
to 2 series of sub-catepories related to a certain methodological questicn), 
or some index score covering the proportion-of didules including certain 
educaticnal features probably resulting in desirable side effects (when some 
of the sab-categories belonging ta a certain methodological question are 
thought to be educationally mere desirable thanithe other sub-groups), 

Such calculetions should net, of course, be used too routinely, nor with 
3 of the scores derived. Nevertheless, 


(c) Gombined protocols 


‘ Combined pretocels register both subject fatter distributicn and 
particular methodological approaches in some kind of combination, In 

many cases it is not enough to see whether the subject matter is suitably 
divided cver the didules, and whether the methods used are sufficiently 
varied, “Ve also want to make sure that the varicus metheds enter at the 
most appropriate points in the subjéct matter sequence, Sometimes we 
consider other methodological approaches adequate when anew concept is 
introduced than when the concept'is already relatively well incorporated 

in the behavior répertoire of the student, ‘The types of prompting techniques 
that. we cre perhaps rcluctant to accept when we deal with a terminology 

that is well known to the student (for instance, prompting of a wholly formal- 
mechanistic type), we may sometimes find acceptable, or even natural, 

when we introduce anew term In addition,, we sometimes want to make 

sure that we do not connect a certain subject matter area too closely with 

a certain method in the student 6 behavioral rgertvire, Net only do we 

want to have an overall methodological variation, but now and then we also 
have to,check that we use a methodologically varied technique in the treat- 
ment of cach single subject matter area. : ~~ 

_ In some respects, the combined protocols are very similar to the 
subject matter protocols: a square-ruled arrangement with didule numbers 
along the top and subject matter units listed down the left-hand margin, 
However, instead of the simple check-marking process, we now use a : 
series of different symbols for various methodological categories (cf Figure 5), 


A final note | 


It showld be mentioned that protocols and diagrams of these:types may 
often-be good training iastruments for persons ‘wenting to train their ability 
to observe educationally important differences between various published 
programy. Hence, in courses about ueclf-instructional materials and pro- 
gramming, protocols of these and vimilar types seem to be well suited to 
mnke the practical trainjny of the study group more effective and efficient, 
Howevery the main aim of these protocols is, of course, to be an aid in the 
process of constructing a program, making the programmer or an indepen- 
dent examiner aware of possible faults in the program uven before the first 
empirical try-out, As stated above, it is hoped that thig will reduce the work 
and cost invelved-in the final phase of successive revision. As our scientific 
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2.!) word-form premipts 


2 2 torield wibphasis ~ 


2.4 syntactic controls 


: i 7 


foe awe, 


poet ee ren es ae a re 


‘ar ‘ . . . 
"2,4 parallel exemplincation 


ee eS ee et 


2.58 induction frum experience | 


et tree te oe eee ee ete 


2.6 jopreal cgnelusivy 


Reading key; fn dednde To the student can obtain the tntormation needed for 
his answer directly from what is said within this didule (“internal-explicit’ | 
Cilitated by strong furimal prompting 


response suurcey, and the answer is ti 
(wurd-form prompts, emphasizing the key words, and svntactie controls). 
te. 


The esample shows a one-sided use of Uinlernal-explicit! 


Comment: 
ses, however, there 


pune suurces ak well as formal prompting. In two evi 
is nv prompting atall, but the student is expected to recall material from 
earlier didules (no. and oo. 10). Tn no case use is made of parallel) oxempli- 
ficwtiony jnduendin from experience, or dog 
iHugtrated bere continues aver a larger material, there is the risk of both 
monotony and less adequ side-efleets (kick of training invapprapriate study 
techniques). : ‘ . : 


J conclusion. PT the one-sidediuess 


is. : 
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; Figure 5 


An example uf a combined protocol (didule protocal, - 


type ©) 


mitter 
units 


Unrel. xX | — i | 
‘ . . 2 


Reading key: 


Symbols for response suurces: ’“Symbols for prompting types: ' 
. 
os = internal-explicit W = word-furm prompts 
(J) = internal+imnplicit M = formal emphasis , 
CJ = extern. «proximal ; , 7 = induction from experience 
soemter al-distal - 1, sloyieal conclusion © r 


it 


parallel exernplitieation 


The protocol is read in the following way: The subject matter unit C, is treated 
in the didules 2, 3, 4 and liter on also in numbers 8, 1] and 15, In didulus’ 

2 and.3 the student obtains his answer within the samo Yidule (first with the 

aid of word-form prornpts, then with the aid of formal emphasis). In the other 
cases with C, the student has to rely on memory. Ete. 


. 


Comment; uN : 

The didule sequence seems to be well constructed. The.iniero sequences po 

from fairly easy tasks (with prompting) to mere difficult ones (without prompting), 
non-forrnal prompting is frequently used, and the sequences contain problems — 
that contribute to the integration of knowledge aa final elements (number 8, 11 


and 15). ; : ; ; 
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knowledge of the self 


vstructional learning process is gradually in- 


creased; the questions studied by means of these checklists, diagrams 


secific, and the evaluation will be more 


empirical ‘try-auyts with subsequent revisions will probably always 
remain an important finel checking and polishing proccss, but the 
amount of work involved in it will decrease, anc at the same time 


t 


"it will be possible and natural te pay increased attention to the pest-" 


writing, pre-testing examination, 


ey 


Stevan TTT 


